A voice verifying system is always applied to a portable electronic apparatus such as a GPS or a mobile phone, such that a user can use a voice command (i.e. a speech signal) to activate the portable electronic device or to control the portable electronic device to perform some function. However, in order to receive the voice, a microphone of the portable electronic device needs to be always turned on to receive the voice. Also, a verifying module of the portable electronic device is always turned on in order to verify the received voice. Therefore, the power consumption for the voice verifying system of related art is high.